Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

patch rexml to improve performance of multi-threaded xml parsing #84

Merged
merged 3 commits into from
Oct 29, 2024
Merged

patch rexml to improve performance of multi-threaded xml parsing #84

merged 3 commits into from
Oct 29, 2024

Conversation

jsvd
Copy link
Member

@jsvd jsvd commented Oct 29, 2024 •
edited
Loading

This change monkey patches the SourceFactory::create_from method from rexml to not require 'stringio'. This is done by copying that method to this plugin and moving the require call to the top.

To check the performance improvement. Before:

❯ time bin/logstash -e 'input { java_generator { message => "<root> <item>&#x1111;&#x1111;&#x1111;&#x1111;&#x1111;&#x1111;&#x1111;&#x1111;&#x1111;&#x1111;</item> <item>&#x2222;&#x2222;&#x2222;&#x2222;&#x2222;&#x2222;&#x2222;&#x2222;&#x2222;&#x2222;</item> <item>&#x3333;&#x3333;&#x3333;&#x3333;&#x3333;&#x3333;&#x3333;&#x3333;&#x3333;&#x3333;</item> <item>&#x4444;&#x4444;&#x4444;&#x4444;&#x4444;&#x4444;&#x4444;&#x4444;&#x4444;&#x4444;</item> <item>&#x5555;&#x5555;&#x5555;&#x5555;&#x5555;&#x5555;&#x5555;&#x5555;&#x5555;&#x5555;</item> <item>&#x6666;&#x6666;&#x6666;&#x6666;&#x6666;&#x6666;&#x6666;&#x6666;&#x6666;&#x6666;</item> </root>" count => 100000 } } filter { xml { target => target source => message } } output { null {} }' --log.level=error
Using bundled JDK: /tmp/test/logstash-8.15.3.slow/jdk.app/Contents/Home
Sending Logstash logs to /tmp/test/logstash-8.15.3.slow/logs which is now configured via log4j2.properties
bin/logstash -e  --log.level=error  100.31s user 25.54s system 242% cpu 51.794 total

After:

❯ time bin/logstash -e 'input { java_generator { message => "<root> <item>&#x1111;&#x1111;&#x1111;&#x1111;&#x1111;&#x1111;&#x1111;&#x1111;&#x1111;&#x1111;</item> <item>&#x2222;&#x2222;&#x2222;&#x2222;&#x2222;&#x2222;&#x2222;&#x2222;&#x2222;&#x2222;</item> <item>&#x3333;&#x3333;&#x3333;&#x3333;&#x3333;&#x3333;&#x3333;&#x3333;&#x3333;&#x3333;</item> <item>&#x4444;&#x4444;&#x4444;&#x4444;&#x4444;&#x4444;&#x4444;&#x4444;&#x4444;&#x4444;</item> <item>&#x5555;&#x5555;&#x5555;&#x5555;&#x5555;&#x5555;&#x5555;&#x5555;&#x5555;&#x5555;</item> <item>&#x6666;&#x6666;&#x6666;&#x6666;&#x6666;&#x6666;&#x6666;&#x6666;&#x6666;&#x6666;</item> </root>" count => 100000 } } filter { xml { target => target source => message } } output { null {} }' --log.level=error
Using bundled JDK: /tmp/test/logstash-8.15.3.fast/jdk.app/Contents/Home
Sending Logstash logs to /tmp/test/logstash-8.15.3.fast/logs which is now configured via log4j2.properties
bin/logstash -e  --log.level=error  121.21s user 2.96s system 682% cpu 18.194 total

closes #83

@jsvd jsvd marked this pull request as ready for review October 29, 2024 15:33
jsvd
jsvd commented Oct 29, 2024
View reviewed changes
CHANGELOG.md Outdated Show resolved Hide resolved
andsel
andsel approved these changes Oct 29, 2024
View reviewed changes
Copy link
Contributor

@andsel andsel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@jsvd jsvd merged commit 8ebd0ba into logstash-plugins:main Oct 29, 2024
2 checks passed
@jsvd jsvd deleted the patch_xml_for_performance branch October 29, 2024 19:06
@yaauie
Copy link
Contributor

yaauie commented Oct 29, 2024

Is this worth upstreaming to ruby/rexml?

@jsvd
Copy link
Member Author

jsvd commented Oct 30, 2024

@yaauie I have submitted a PR upstream but I don't have a lot of hope there since it's a JRuby specific thing (maybe less so now with Ractors in Ruby 3.x) ruby/rexml#219

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@andsel andsel andsel approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Slow store_xml performance due to rexml performing one "require" on per event
3 participants
@jsvd @yaauie @andsel